System and method for finding regions of interest for microscopic digital montage imaging

ABSTRACT

A system for determining tissue locations on a slide.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.09/758,037, filed Jan. 11, 2001, now U.S. Pat. No. 6,993,169 which isincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to microscopic digital imaging of tissuesections for medical and research use. In particular it describes amethod to find regions of interest for high throughput montage imagingof microscope slides using a standard microscope and camera.

BACKGROUND OF THE INVENTION

Laboratories in many biomedical specialties, such as anatomic pathology,hematology, and microbiology, examine tissue under a microscope for thepresence and the nature of disease. In recent years, these laboratorieshave shown a growing interest in microscopic digital imaging as anadjunct to direct visual examination. Digital imaging has a number ofadvantages including the ability to document disease, share findings,collaborate (as in telemedicine), and analyze morphologic findings bycomputer. Though numerous studies have shown that digital image qualityis acceptable for most clinical and research use, some aspects ofmicroscopic digital imaging are limited in application.

Perhaps the most important limitation to microscopic digital imaging isa “subsampling” problem encountered in all single frame images. Thesub-sampling problem has two components: a field of view problem and aresolution-based problem. The field of view problem occurs when aninvestigator looking at a single frame cannot determine what liesoutside the view of an image on a slide. The resolution-based problemoccurs when the investigator looking at an image is limited to theresolution of the image. The investigator cannot “zoom in” for a closerexamination or “zoom out” for a bird's eye view. Significantly, thefield of view and resolution-based problems are inversely related. Thus,as one increases magnification to improve resolution, one decreases thefield of view. For example, as a general rule, increasing magnificationby a factor of two decreases the field of view by a factor of four.

To get around the limitations of single frame imaging, developers havelooked at two general options. The first option takes the general formof “dynamic-robotic” imaging, in which a video camera on the microscopetransmits close to real time images to the investigator looking at amonitor, while the investigator operates the microscope by remotecontrol. Though such systems have been used successfully fortelepathology, they do not lend themselves to documentation,collaboration, or computer based analysis.

The second option being investigated to overcome the limitations inheritin single frame imaging is a montage (or “virtual slide”) approach. Inthis method, a robotic microscope systematically scans the entire slide,taking an image at every field. The individual images are then “knitted”together in a software application to form a very large data set withvery appealing properties. The robotic microscope can span the entireslide area at a resolution limited only by the power of the opticalsystem and camera. Software exists to display this data set at anyresolution on a computer screen, allowing the user to zoom in, zoom out,and pan around the data set as if using a physical microscope. The dataset can be stored for documentation, shared over the Internet, oranalyzed by computer programs.

The “virtual slide” option has some limitations, however. One of thelimitations is file size. For an average tissue section, the datagenerated at 0.33 um/pixel can be between two and five gigabytesuncompressed. In an extreme case, the data generated from one slide canbe up to thirty-six gigabytes.

A much more difficult limitation with the prior systems is an imagecapture time problem. Given an optical primary magnification of twentyand a two-third inch CCD, the system field of view is approximately (8.8mm.times.6.6 mm)/20=0.44.times.0.33 mm. A standard microscope slidetypically has a specimen area of 25 mm.times.50 mm or 12.5 squarecentimeters. This requires over eighty-six hundred fields to image thisentire specimen region. However, the average tissue section for anatomicpathology is approximately 2.25 square centimeters. This only requiresapproximately fifteen hundred fields to cover the tissue alone,approximately 80 percent less fields.

Traditionally, field rate in montage systems is limited by threefactors—camera frame rate, image processing speed, and the rate of slidemotion between fields. Given today's technology, the limiting factor canbe reduced to only the camera frame rate. Using a 10 frame per secondcamera for the example above, imaging the entire slide would require 860seconds or 14.33 minutes. If only the region of interest was imaged,this average time could be reduced to 150 seconds or 2.5 minutes;substantially increasing the slide throughput of an imaging system.

Thus, a system is needed to automatically find the region of interest ona microscope slide and image only this region.

SUMMARY OF THE INVENTION

An embodiment of a tissue finding method disclosed herein includesdifferentiating tissue containing regions of a microscope slide fromnon-tissue containing regions of the microscope slide. An embodiment ofa tissue finding apparatus disclosed herein includes a device todifferentiate tissue containing regions of a microscope slide fromnon-tissue containing regions of the microscope slide.

The present invention relates to a method and system for processing athumbnail image from a microscope slide to determine tissue locations onthe slide. An embodiment of the system comprises an image croppingcomponent, a tissue finding component, and a scan control component. Theimage cropping component crops the thumbnail image and removes portionsof the image that fall outside of determined slide boundaries. Thecropped image from the image cropping component is inputted into thetissue finding component. The tissue finding component identifies tissueregions by applying a sequence of filters that incorporate knowledge oftypical appearance and location of tissue and non-tissue slide regions.The tissue finding component outputs a tiling matrix whose valuesindicate which tiles should be imaged. The scan control componentinterprets the tiling matrix and transposes positions of the tilingmatrix into actual stage coordinate for a microscopic imaging.

Accordingly, it is an object of an embodiment of the invention toprovide a microscopic imaging system for whole slide montage in whichstandard microscope optics, off the shelf cameras and a simple motorizedstage can be used to select the region of interest, image only thissection and produce aligned image tiles.

An embodiment of the present invention uses a pre-scan process appliedto a macroscopic image of the slide, to guide a high-resolution slidescanning process and ensure high-quality images of the entire specimenare acquired. The pre-scan process includes an image cropping component,a tissue-finding component, and a scan control component. The imagecropping and tissue finding components identify interesting regions onthe slide to be scanned. The scan control component generates thecontrol parameters for a motorized microscopic imaging system.

It is another object of an embodiment of the invention to use ahigh-resolution slide scanning process to control the operation of themotorized stage and camera. This process utilizes information gatheredby the pre-scan process, namely the imaging regions, to control thepositioning of the stage to image only the regions of interest and toensure the individual images are well aligned.

Additional features and advantages of the invention will be set forth inthe description that follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. Theobjectives and advantages of the invention to be realized and attainedby the microscopic image capture system will be pointed out in thewritten description and claims hereof as well as the appended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention thattogether with the description serve to explain the principles of theinvention.

FIG. 1 illustrates an isometric view of the system in an embodiment;

FIG. 2 represents sample results of the macroscopic image after thecropping component has been applied to remove non-slide regions;

FIG. 3 represents sample results of the find tissue component; and

FIG. 4 is an overlay of FIGS. 2 and 3 representing the regions of theslide to be imaged.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. The following paragraphs describe the functionality of theinventive system and method for high throughput montage imaging ofmicroscope slides using a standard microscope and cameras.

FIG. 1 illustrates an embodiment of the invention. In this embodiment, aslide 112 to be imaged is placed on a thumbnail imaging position in aslide holder on a motorized stage 102. A single frame image containingthe entire slide is taken with a macro camera 106. This low-resolutionimage is analyzed by software components to determine the locations oftissue on slide 112. This information can then be used to generatecontrol parameters for stage 102 and microscopic camera 104 to ensurethat the scanning process captures high quality images of only thetissue regions, substantially reducing the time to scan an averageslide.

As is obvious to one skilled in the art, although capturing the singlemacroscopic image saves time, it is not necessary for the operation ofthe invention. Multiple macroscopic images may be required to generatecontrol parameters to the accuracy required based on the ratio of themacroscopic to microscopic magnifications and the camera specificationsof each camera, if separate cameras are utilized.

Specifically in an embodiment, a pre-scan processing of thelow-resolution or thumbnail image includes an image cropping component,a tissue-finding component and a scan control component. The imagecropping component and tissue finding component identify tissue regionson the slide to be scanned. The scan control component generates thenecessary control parameters to scan only the regions of interest underthe microscopic optics.

The first step in processing the thumbnail image may consist offlat-field correcting the macroscopic thumbnail image using a similarimage obtained from the same camera and a blank slide. This removes anyspatial light anomalies from the thumbnail image, which may reduce theefficiency of the tissue-finding component. Given the format, or size,of the camera and the aspect ratio of the slide, a portion of the imagewill contain non-slide objects such as the slide carrier. To removethese features, the thumbnail image may be cropped to extract only theslide information.

The image cropping may be accomplished via a two-pass process. The firstpass determines an approximate location of the slide boundary, and thesecond pass fine-tunes this estimate. The search for the boundary may beconducted over upper and lower intervals corresponding to the regionsexpected to contain the upper and lower slide edges, respectively. Forthis discussion, the slide or region of interest is assumed to bepositioned near the center, vertically, in the thumbnail image. Tofacilitate this and subsequent processing steps, a copy of the thumbnailimage may be converted to grayscale. The portion of the image fallingoutside of the identified slide boundary may be removed. It should benoted that the original color image may also be cropped at the estimatededge locations, and then uniformly reduced in size to produce a smallthumbnail image of the slide for rapid, visual slide identification.

Since the slide may not be oriented perfectly horizontal in the originalthumbnail image, the identified slide edges are likely to lie at anangle. Thus, even after cropping, there may be remnants of the slideedges or cover slip in the cropped image. Therefore, the image-croppingcomponent attempts to identify pixel blocks that likely contain theseremaining edges and flags these blocks as edges that will not beconsidered for high resolution imaging by the tissue finding component.

The resulting cropped grayscale image generated by the image-croppingcomponent serves as input to the tissue finding component. Thiscomponent locates regions in the thumbnail image that contain tissue ofinterest to a specialist. In order to minimize the time and storagespace required to accomplish high-resolution slide imaging, theinventive system captures only those regions of the slide that containtissue. Embodiments of this approach may require that regions containingtissue be identified in the thumbnail image.

The tissue finding component identifies tissue regions via a sequence offilters that incorporate knowledge of the typical appearance andlocation of tissue and non-tissue slide regions. Initial filtering stepsmay analyze the mean and standard deviation of the local pixelintensities. Pixel mean intensities may be used to differentiatetissue-containing regions from blank and other non-tissue regions, suchas those containing the slide label or other markings. The standarddeviation data may represent the amount of variation in pixel values andthus is a good indicator of the border between tissue and the blankslide. The mean and standard deviation data may be combined to generatea threshold value that is used to make an initial classification oftissue versus non-tissue. Subsequently, morphological filters may beapplied to refine the classification based on the size and position ofneighboring groups of potential tissue pixels.

The filters which comprise the tissue finding component process thepixels of the cropped grayscale thumbnail image in groups thatcorrespond to slide regions, or tiles, that can be imaged individuallyduring the high-resolution scanning process. These filters ensure thattiles only partially filled with tissue are classified astissue-containing tiles. The final output of the filter sequence is atiling matrix whose values indicate which tiles should be imaged; thetiling matrix subsequently guides the high-resolution scanning process.

The above description was based on using the mean and standard deviationof the local pixels as the basis for detecting regions of interest. Itis obvious to one skilled in the art that other image characteristicscan be also used to identify the specimen from non-items of interestsuch as dust and scratches.

This description was also based on processing a gray scale macroscopicimage, the same processing tools can be applied to each of the colorcomponents (traditionally, red, green and blue) of a color image.Additional processing tools can also be applied between the colorcomponents to refine the tissue finding accuracy and to remove featuressuch as labels and writing that are not critical to the application.

An example of the image cropping and find tissue processing is shown inFIGS. 2, 3 and 4. FIG. 2 illustrates the macroscopic image afterflat-field correction and image cropping. FIG. 3 illustrates the resultsof the find tissue component. The resulting tile matrix shown in FIG. 3has a one-to-one correspondence to the field of view of the microscopiccamera. White pixels, which may be associated with a binary 1, maysignify fields to be capture and black pixels may represent regions notto image. FIG. 4 illustrates an overlay FIGS. 2 and 3 representing thesections of the slide to be imaged. For this application (anatomicalpathology), it may be imperative to image all suspect regions that maycontain tissue, so conservative criteria were used in the find tissuecomponent, resulting in cover slip edges and writing etched into theslide to be identified as to be imaged. The savings in the acquisitiontime is representative by the ratio of the white to black areas of FIG.3. For this image, only 53% of the slide region is to be imaged,including the label and cover slip edges, and etched writing on theslide.

At the completion of the find tissue component, in this example, thescan control component interprets the find tissue tile matrix (FIG. 3)and transposes the positions into actual stage coordinates for themicroscopic imaging. A program running on a host computer controls theoperation by communicating with a stage controller and microscopiccamera 104. Actual scanning can occur in any fashion such as by rows orcolumns, or in a step fashion to image neighboring areas.

The foregoing description has been directed to specific embodiments ofthis invention. It will be apparent, however, that other variations andmodifications may be made to the described embodiments, with theattainment of some or all of their advantages. Therefore, it is theobject of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of the invention.

1. A system for determining a specimen location on a slide and capturingan image of that location, the system comprising: a camera taking a lowresolution image of the slide; a first component for automaticallyidentifying a region of the low resolution image containing thespecimen, where said region defines only a portion of the slide; and animage capture component to generate control parameters to controlmovement of a motorized stage and operation of the camera to capture animage of only the identified region through microscopic optics; whereinthe low resolution image is inputted into a tissue finding component,and wherein the tissue finding component identifies one or more tissueregions in the low resolution image by applying a filter thatincorporates knowledge of typical appearance and location of tissue andnon-tissue slide regions and outputs a matrix having values thatindicate which slide regions are of interest; and wherein the filterconverts a copy of the low resolution image to grayscale and analyzes atleast one of mean and standard deviation of local pixel intensities inthe grayscale image to generate a threshold value for classifying tissueversus non-tissue regions.
 2. The system of claim 1, wherein the firstcomponents further comprises: an image cropping component fordetermining a location of a slide boundary by searching upper and lowerintervals corresponding to boundary regions expected to contain upperand lower edges of the slide and cropping portions of the image fallingoutside of the determined slide boundary location.
 3. The system ofclaim 2, wherein the low resolution image is a color image and the imagecropping component crops the color image at the slide boundary.
 4. Thesystem of claim 2, wherein the image cropping component reduces the lowresolution image size to produce a small thumbnail image of the specimenfor rapid visual identification.
 5. The system of claim 2, wherein theimage cropping component identifies pixel blocks in the cropped imagethat are likely to contain remaining slide edge features and flags theblocks as edges that should not be considered for high-resolutionimaging.
 6. The system of claim 1, wherein the first component includesa tissue finding component that locates regions in the low resolutionimage that contain tissue of interest.
 7. The system of claim 1, whereinthe standard deviation represents the amount of variation in pixelintensity and is a good indicator of the border between tissue regionsand blank regions of the slide.
 8. The system of claim 1, wherein amorphological filters are applied to the matrix to identify slideregions that can be imaged individually during a high-resolution imagingprocess.
 9. The system of claim 8, wherein the morphological filtersensures that tiles that contain both tissue and non-tissue pixels areclassified as tissue-containing tiles.
 10. The system of claim 1,wherein the low resolution image is taken as a single image.
 11. Thesystem of claim 1, wherein the low resolution image is taken as multipleimages.
 12. The system of claim 1, wherein the first component includesan image capture control component that interprets the identifiedregions into actual microscope stage coordinates for a microscopicimaging.
 13. The system of claim 1, wherein the low resolution image istaken automatically without human intervention, the specimen containingregion is identified automatically without human intervention, and thecontrol parameters are generated automatically without humanintervention.
 14. The system of claim 1, wherein the components aresoftware components executed by a computer.
 15. A method for processinga low resolution image from a slide to determine a specimen location onthe slide, the method comprising: cropping the low resolution image toremove portions of the low resolution image that correspond to non-slideobjects; inputting the cropped image into a tissue finding component,wherein the tissue finding component identifies a region containing thespecimen by applying a filter that incorporates knowledge of typicalappearance and location of specimen and non-specimen slide regions andoutputs a matrix whose values indicate which regions of the slide shouldbe imaged; converting a copy of the low resolution image to a grayscaleimage and analyzing at least one of mean and standard deviation of localpixel intensities in the grayscale image to generate a threshold valuefor classifying tissue versus non-tissue regions; and transposingpositions of the matrix into actual stage coordinates, and capturing amicroscopic image at those stage coordinates.
 16. A system forprocessing a low resolution image from a microscope slide to determinetissue locations on the slide, the system comprising: a camera takingthe low resolution image from the slide; an image cropping component forcropping non-slide objects from the low resolution image; a tissuefinding component that identifies a region containing a tissue in thelow resolution image by applying a sequence of filters that incorporateknowledge of typical appearance and location of tissue and non-tissueslide regions and outputs a matrix whose values indicate which tissueportions of the low resolution image should be imaged; a scan controlcomponent for interpreting the matrix; and a controller, inputting acropped image from the image cropping component into the tissue findingcomponent and transposing positions of the matrix into actual stagecoordinates for microscopic imaging; wherein the low resolution image isinputted into a tissue finding component, and wherein the tissue findingcomponent identifies one or more tissue regions in the low resolutionimage by applying a filter that incorporates knowledge of typicalappearance and location of tissue and non-tissue slide regions andoutputs a matrix having values that indicate which slide regions are ofinterest; and wherein the filter converts a copy of the low resolutionimage to grayscale and analyzes at least one of mean and standarddeviation of local pixel intensities in the grayscale image to generatea threshold value for classifying tissue versus non-tissue regions. 17.A tissue finding method, for differentiating tissue containing regionsof a microscope slide from non-tissue containing regions of themicroscope slide, comprising: Taking a low resolution image of theslide; automatically identifying a region of the low resolution imagecontaining the specimen, where said region defines only a portion of theslide; generating control parameters to control movement of a motorizedstage and operation of the camera to capture an image of only theidentified region through microscopic optics; inputting the lowresolution image into a tissue finding component, and identifying one ormore tissue regions in the low resolution image by applying a filterthat incorporates knowledge of typical appearance and location of tissueand non-tissue slide regions and outputs a matrix having values thatindicate which slide regions are of interest; and converting a copy ofthe low resolution image to grayscale and analyzing at least one of meanand standard deviation of local pixel intensities in the grayscale imageto generate a threshold value for classifying tissue versus non-tissueregions.
 18. The tissue finding method of claim 17, further comprisingcapturing a magnified image of each tissue containing region.
 19. Thetissue finding method of claim 17, further comprising: capturing animage of at least a portion of the microscope slide; and grouping pixelsof the captured image into regions.
 20. The tissue finding method ofclaim 17, wherein regions only partially filled with tissue areclassified as tissue containing.
 21. The tissue finding method of claim17, further comprising providing a tiling matrix having values thatindicate which regions should be imaged.
 22. The tissue finding methodof claim 21, further comprising transposing the tiling matrix intocoordinates of an image to be captured.
 23. A tissue finding apparatusthat is to group pixels comprising an image of a microscope slide intoregions and differentiate the regions that contain at least a portion ofan image of a tissue from the regions that do not contain at least aportion of an image of a tissue; wherein a low resolution image isinputted into the tissue finding apparatus, and wherein the tissuefinding apparatus identifies one or more tissue regions in the lowresolution image by applying a filter that incorporates knowledge oftypical appearance and location of tissue and non-tissue slide regionsand outputs a matrix having values that indicate which slide regions areof interest; and wherein the filter converts a copy of the lowresolution image to grayscale and analyzes at least one of mean andstandard deviation of local pixel intensities in the grayscale image togenerate a threshold value for classifying tissue versus non-tissueregions.
 24. The tissue finding apparatus of claim 23, furthercomprising: a stage on which the microscope slide is to be placed; amicroscopic optic directed toward the stage; and a camera directedtoward the stage and coupled to the tissue finding apparatus.
 25. Thetissue finding apparatus of claim 24, wherein the camera is to capturean image of at least a portion of the microscope slide and the tissuefinding apparatus is further to capture an image of each regioncontaining at least a portion of an image of a tissue using the cameradirected through the microscopic optic.
 26. The tissue finding apparatusof claim 23, wherein the tissue finding apparatus is to differentiatetissue containing regions of the microscope slide from non-tissuecontaining regions of the microscope slide based, at least in part, onthe typical appearance of tissue and non-tissue regions.
 27. The tissuefinding apparatus of claim 23, wherein the tissue finding apparatus isto differentiate tissue containing regions of the microscope slide fromnon-tissue containing regions of the microscope slide based, at least inpart, on the typical location of tissue and non-tissue regions.
 28. Thetissue finding apparatus of claim 23, wherein the tissue findingapparatus is further to provide a tiling matrix having values thatindicate which regions should be imaged.
 29. The tissue findingapparatus of claim 28, wherein the tissue finding apparatus is furtherto transpose the tiling matrix into coordinates of an image to becaptured.
 30. A tissue finding apparatus that is to differentiate tissuecontaining regions of a microscope slide from non-tissue containingregions of the microscope slide wherein a low resolution image isinputted into the tissue finding apparatus, and wherein the tissuefinding apparatus identifies one or more tissue regions in the lowresolution image by applying a filter that incorporates knowledge oftypical appearance and location of tissue and non-tissue slide regionsand outputs a matrix having values that indicate which slide regions areof interest; and wherein the filter converts a copy of the lowresolution image to grayscale and analyzes at least one of mean andstandard deviation of local pixel intensities in the grayscale image togenerate a threshold value for classifying tissue versus non-tissueregions.
 31. The tissue finding apparatus of claim 30, furthercomprising: a stage on which the microscope slide is to be placed; amicroscopic optic directed toward the stage; and a camera directedtoward the stage and coupled to the tissue finding apparatus.
 32. Thetissue finding apparatus of claim 31, wherein the camera is to capturean image of at least a portion of the microscope slide and the tissuefinding apparatus is further to capture an image of each regioncontaining at least a portion of an image of a tissue using the cameradirected through the microscopic optic.
 33. The tissue finding apparatusof claim 30, wherein the tissue finding apparatus is further todifferentiate tissue containing regions of the microscope slide fromnon-tissue containing regions of the microscope slide based, at least inpart, on the typical appearance of tissue and non-tissue regions. 34.The tissue finding apparatus of claim 30, wherein the tissue findingapparatus is further to differentiate tissue containing regions of themicroscopic slide from non-tissue containing regions of the microscopeslide based, at least in part, on the typical location of tissue andnon-tissue regions.
 35. The tissue finding apparatus of claim 30,wherein the tissue finding apparatus is further to provide a tilingmatrix having values that indicate which regions should be imaged. 36.The tissue finding apparatus of claim 35, wherein the tissue findingapparatus is further to transpose the tiling matrix into coordinates ofan image to be captured.